Method for Correcting an Acquired Image

ABSTRACT

A method of correcting an image obtained by an image acquisition device includes obtaining successive measurements, G n , of device movement during exposure of each row of an image. An integration range, idx, is selected in proportion to an exposure time, t e , for each row of the image. Accumulated measurements, C n , of device movement for each row of an image are averaged across the integration range to provide successive filtered measurements,  G , of device movement during exposure of each row of an image. The image is corrected for device movement using the filtered measurements  G .

RELATED APPLICATION

The present application relates to co-filed application Ser. No.15/048,149 entitled “A Method of Stabilizing a Sequence of Images”,filed Feb. 19, 2016, the disclosure of which is incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to a method for correcting an acquiredimage.

BACKGROUND

Referring to FIG. 1, image acquisition devices 10 typically comprise animage processing pipeline (IPP) 12 which obtains acquired image datafrom an image sensor (not shown), conducts basic processing of the rawimage data, such as color balancing, and writes acquired images orportions of images via a system bus 20 to system memory 14. Image framescan be acquired by the IPP 12 at frame rates from 24 fps up to from 60fps to even 240 fps.

Such image acquisition devices 10 can include downstream dedicated imageprocessing units which can analyse acquired images and process suchimages either to extract information from the images or to correct theimages. Such processing can includes face detection and tracking, objectrecognition or distortion correction such as disclosed in PCTApplication WO2014/005783 (Ref: FN-384). In the present specification,such processing units, which can be dedicated hardware modules or ageneric central processing unit (CPU), are indicated as processing unit(PU) 16 which is capable of running either low-level firmware/softwareor in the case of the CPU, application software, capable of obtainingimage information from memory 14 and further processing the images.

In the present specification, we refer to images being provided by IPP12, however, it will be appreciated that these can comprise eitherindividually acquired images or images within a video sequence.

It is known for image acquisition devices 10 to include inertialmeasurement units (IMU) 18 which record a trajectory of device movementduring image acquisition, enabling processing unit(s) 16 to use thatinformation to correct an acquired image to take into account blurcaused by involuntary or unwanted device motion during image capture orto stabilize video sequences, for example, as disclosed in Sung Hee &Park Marc Levoy “Gyro-Based Multi-Image Deconvolution for RemovingHandshake Blur”, pp 3366-3373, Conference on Computer Vision and PatternRecognition, 23-28 Jun. 2014. IMUs 18 can include any combination ofgyroscopes, accelerometers and magnetometers. Typically involuntary orunwanted device motion blur is caused by human hand shake or tremor asdistinct from a deliberate sweeping movement used by photographers orvideographers when attempting either to simulate motion blur or tocapture an image of a moving object.

Referring to FIG. 2(a), some image acquisition devices employ a rollingshutter technique where one or more lines (rows) of an image are readfrom an image sensor during successive exposure intervals. Each line orgroup of lines R₁,R₂,R₃,R₄ . . . of the image sensor is exposed overcertain period of time t_(I1), meaning the light intensity is integratedover that period of time. If the acquisition device is moved during theexposure time, the total line exposure is a sum of light coming fromdifferent directions. If an image acquisition device is subjected torelatively high frequency vibrational movement during image acquisitionusing rolling shutter, i.e. a frequency (f) higher than the totalexposure time for the image frame t_(f), then a vertical straight edgewithin a scene can appear to oscillate as indicated by the line 22.

Let us first assume a perfect synchronization between IMU samples andthe starts of exposure for each of the (groups of) line(s) R₁,R₂,R₃,R₄ .. . of the image sensor. When the exposure time is very short, i.e.where the frequency of vibration (f) is less than 1/t_(I1), where t_(I1)is exposure time for a line, the captured image represents a shortmoment in time for a given camera orientation. In that case, motion ofimage features correlates very well with the motion recorded by the IMUsensors during the exposure time for a given (group of) line(s)—eachbeing generally linear or at least monotonic. Thus, by delaying an IMUsignal by ½ of the exposure time t_(I1), such an image can correctedaccording to the IMU sensor data.

However, with the increasing exposure time, t_(I2)>t_(I1), such acorrection becomes inappropriate. So referring to FIG. 2(b), a verticalline captured with a rolling shutter where vibration is at a frequency(f) higher than 1/t_(I2), without correction, appears as a blurred line24 within an image. Correction performed according to the IMU sensordata now starts to introduce overcompensation, which in the extrememakes the corrected result look worse, by further distorting the line24, pulling alternating segments of the line in opposite directionswithout unblurring the line detail.

So while relatively slower (lower frequency) camera movements can stillcompensated properly even with a long exposure image wheret_(I2)>t_(I1), if the camera is vibrating at frequencies higher than1/t, the exposure time for lines of the image, overcompensation isparticularly visible.

This becomes a particular problem in drone mounted cameras where thevibration caused by the drone rotors can be of the order of 100 Hz. Forimages acquired at high light levels, exposure times of as little as 1ms may be employed and so conventional correction of these images isunaffected by the vibration. However, for images acquired in lower lightenvironments, exposure levels of up to 30 ms or so may be required andthese images can be heavily distorted if corrected using conventionaltechniques.

It is tempting to address the above problem by making correctionamplitude a direct function of exposure time, i.e. to damp the degree ofcorrection in proportion to exposure time. However, this does notprovide a satisfactory solution, as this affects all vibrationfrequencies in the same way and so for example, correction oflonger-exposed images subject to low frequency shake becomesparticularly unsatisfactory.

SUMMARY

According to the present invention, there is provided a method ofcorrecting an image according to claim 1.

Embodiments filter data provided by an inertial measurement unit toprovide for stabilization of images with varying exposure times whichcan be subject to vibration frequencies higher than the readoutfrequency of an image sensor.

The invention is based on the observation that for longer exposure timeimages, mechanical vibrations, typically greater than human hand shakeor tremor frequencies and for example caused by drone motors, causeoscillating motion blur within acquired images, including images withina video sequence.

Correction of such longer exposure time images which are subject to highfrequency vibration using the embodiment of the present invention doesnot necessarily remove high frequency motion blur from the image, but itprevents further distortion of the image.

The same results cannot be achieved using low-pass filters typicallybuilt into gyroscopes within device IMUs. These are meant to prevent anyaliasing artefacts caused by sampling and to reduce signal noise.Typically they offer several pre-defined filter settings forapplications like inertial navigation or applications which depend ondevice orientation where the frequency components of a measured signalare relatively low. Those filters are implemented as IIR filters with aphase delay that is uncorrelated with the exposure time and can bedifferent for different signal frequencies.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the invention will now be described, by way of example,with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of an image acquisition device according to anembodiment of the invention; and

FIG. 2(a) and FIG. 2(b) illustrate the effect of acquisition devicemovement during exposure of respective images from a rolling shutterexposure.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

In embodiments of the present invention, IMU sensor data is integratedover the exposure time of the lines of an acquired image so tending toaverage the record of device movement. As a result, correction to eachline of an image is calculated for average camera orientation during theexposure time of the line. Importantly, the range of the integrator (orthe corner frequency of the integrator) is a function of the exposuretime of the image.

In one embodiment the IMU sensor data comprises gyroscope samples andthe method operates as follows:

-   -   1. Successive gyroscope samples G_(n) are added to a cumulative        buffer, where each new element is a sum of the current value        G_(n) and all previous sample values added to the buffer. The        cumulative value corresponding to sample G_(n) is denoted as        C_(n)    -   2. An index to the cumulative buffer is calculated dependent on        the gyroscope sampling frequency F and exposure time t_(e) as        follows: idx=F*t_(e)    -   3. An average gyroscope rate is calculated:        G=(C_(n)−C_(n−idx))/idx    -   4. The average rate G is used as the numerical integration of        camera orientation. It is these filtered values G which are used        subsequently in place of the trajectory of corresponding        original samples G_(n) for correction of corresponding lines of        an image in an otherwise conventional fashion. It will be seen        that this mode of filtering does not require any additional        corrections and will automatically adapt to changing exposure        time.

It will be seen that for short exposure images, idx will be short and sothe linear approximation G of device movement during exposure time willprovide similar values to the original samples G and so will beappropriate for correcting such images. On the other hand, as exposuretimes increase, the averaging will have the effect of not overcorrectingan image subject to high frequency vibration, but can still provideuseful correction for images subjected to human hand shake or tremor.

While the above example has been described in terms of gyroscopesamples, the same technique can be applied to sample values from all IMUsensors (gyroscope, accelerometer, magnetometer) in implementationswhere full sensor fusion is required.

The above approach can be employed whether images are acquired using arolling shutter technique or not. Where lines of an image are exposedsuccessively, then successive corresponding IMU measurements filtered asabove can be employed to correct for device movement during exposure ofthose lines; whereas for an image where all lines are exposed at thesame time, then the same IMU measurements filtered as above are appliedto all lines.

The filtered IMU sensor signals described above can be employed inelectronic image stabilisation (EIS) schemes such as disclosed inco-filed application Ser. No. 15/048,149 in place of raw sensor signalsconventionally employed to mitigate problems caused by high frequencyvibration of the camera during image capture.

1. A method of correcting an image obtained by an image acquisitiondevice including: obtaining successive measurements, G_(n), of devicemovement during exposure of each row of an image; selecting anintegration range, idx, in proportion to an exposure time, t_(e), foreach row of the image, averaging accumulated measurements, C_(n), ofdevice movement for each row of an image across said integration rangeto provide successive filtered measurements, G, of device movementduring exposure of each row of an image, and correcting said image fordevice movement using said filtered measurements G.
 2. A methodaccording to claim 1 wherein said integration range, idx, is calculateddependent on a sampling frequency F for said measurements of devicemovement and said exposure time t_(e) as follows: idx=F*t_(e)
 3. Amethod according to claim 1 wherein said averaging comprises calculatingsaid filtered measurements G as follows: G=(C_(n)−C_(n−idx))/idx whereC_(n) is an accumulated value corresponding to measurement G_(n) andC_(n−idx) is an accumulated value corresponding to measurement G_(n−idx)at the start of said integration range.
 4. A method according to claim 1wherein said measurements of device movement comprise any of: gyroscope,accelerometer or magnetometer measurements.
 5. A method according toclaim 1 wherein said image comprises a still image or an image within asequence of images.
 6. A method according to claim 1 wherein said rowsof said image are acquired during successive exposure times, t_(I), saidmeasurements of device movement corresponding to respective exposuretimes.
 7. A processing unit for an image acquisition device arranged tocorrect an image according to the method of claim 1.